Preventing motion artifacts by intelligently disabling video stabilization

ABSTRACT

Digital video stabilization is selectively turned off in circumstances where it could actually decrease the quality of a captured video. A video camera includes a device for directly detecting physical motion of the camera. Motion data from the motion detector are analyzed to see if video stabilization is appropriate. If the motion data indicate that the video camera is stable, for example, then video stabilization is not applied to the video, thus preventing the possibility of introducing “motion artifacts” into the captured video. In another example, motion as detected by the motion detector can be compared with motion as detected by the video-stabilization engine. If the two motions disagree significantly, then the video-stabilization engine is probably responding more to motion in the captured video rather than to motion of the camera itself, and video stabilization should probably not be applied to the video.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is related to U.S. Patent Application (MotorolaDocket Number CS39994), filed on an even date herewith.

TECHNICAL FIELD

The present disclosure is related generally to digital-image captureand, more particularly, to digital video stabilization.

BACKGROUND

Digital video-capture devices, such as still and video cameras, arebecoming more commonplace. Once available only on high-end professionalcameras, today such digital devices are found in smartphones, tabletcomputers, and other personal electronics devices. Digital cameras aresmall and relatively inexpensive to manufacture and are thus popularaccessories in today's personal electronics devices.

Because these personal electronics devices are often quite small andlight, they can be easily shaken when the user is trying to take aphotograph or a movie, and this shaking can produce blurry images. Tocompensate for shaking, digital video stabilization methods comparecaptured images frame by frame and apply global motion compensation toremove the frame-to-frame motion and to thus make the captured imagesless blurry.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

While the appended claims set forth the features of the presenttechniques with particularity, these techniques, together with theirobjects and advantages, may be best understood from the followingdetailed description taken in conjunction with the accompanying drawingsof which:

FIG. 1 is an schematic of a representative personal electronics devicewith a video camera in which the present techniques may be practiced;

FIG. 2 is a flowchart of a representative method for disabling videostabilization when a camera is steady;

FIGS. 3 a through 3 c are graphs of motion-data measurements made for acamera in different circumstances; and

FIGS. 4 and 5 are flowcharts of methods for selectively applying videostabilization when a camera may be moving.

DETAILED DESCRIPTION

Turning to the drawings, wherein like reference numerals refer to likeelements, techniques of the present disclosure are illustrated as beingimplemented in a suitable environment. The following description isbased on embodiments of the claims and should not be taken as limitingthe claims with regard to alternative embodiments that are notexplicitly described herein.

Digital video stabilization is selectively turned off in circumstanceswhere it could actually decrease the quality of a captured video. Avideo camera includes a motion detector device for directly detectingphysical motion of the camera (e.g., a gyroscope, accelerometer, a GPSreceiver, or another camera). Motion data from the motion detector areanalyzed to see if video stabilization is appropriate. If the motiondata indicate that the video camera is stable, for example, then videostabilization is not applied to the video, thus preventing thepossibility of introducing “motion artifacts” into the captured video.In another example, motion as detected by the motion detector can becompared with motion as detected by the video-stabilization engine. Ifthe two motions disagree significantly, then the video-stabilizationengine is probably responding more strongly to motion in the capturedimages than to motion of the camera itself, and video stabilizationshould probably not be applied to the video.

Various embodiments can be implemented in hardware or as executable coderunning on a microprocessor or a controller of an video-capture system.Where so configured, the executable code can include various modulesthat determine video stability and physical motion.

FIG. 1 shows a representative personal electronics device 100 that canembody aspects of the present disclosure. The device 100 can be justabout any device with a video camera 102 including, for example, adedicated camera, a mobile telephone, a personal digital assistant, or atablet computer.

The video camera 102 of FIG. 1 represents an entire video-capturedevice, as known in the art. A typical video-capture device 102 includesan image sensor and a control circuit (represented in FIG. 1 by theprocessor 106, but see the discussion below). The image sensor, whichcan be any type of image sensor known in the art, such as a CMOS or aCCD sensor, receives light through a lens. The light is reflected off asubject and is then captured by the image sensor to form an image. Thecaptured image is stored as one digital frame 104. Generally speaking,the video camera 102 creates frames of captured images as a constantrate. When displayed to a user at the same rate, the succession offrames is a video clip. As is known in the art, the video-capture device102 may include other optical, mechanical, and electronic systems(including a variable aperture, lenses, an auto-focus engine, and thelike). The personal electronics device 100 can include another videocamera (not shown in FIG. 1), the possible use of which is discussedbelow.

For ease of discussion, it is assumed that the personal electronicsdevice 100 is completely controlled by a software-driven processor 106.In some embodiments, however, a microcontroller, embedded controller, orapplication-specific controller 106 controls certain aspects of thedevice 100 including the video stabilizer 108 (also called avideo-stabilization engine) and the motion detector 110 as describedbelow in reference to FIGS. 2, 4, and 5. In some embodiments, controlfunctions are divided among multiple modules within the device 100including both software-controlled processors and hardwired controllers.

The video-stabilization engine 108, as is well known in the art, reviewsthe captured frames 104 and attempts to detect motion of the personalelectronics device 100. If, for example, a pixel-by-pixel comparison oftwo successive frames 104 reveals that the second frame is essentiallyidentical to the first but displaced by a small but constant amount tothe right, then it is possible that the image being captured has notmoved to the right, but instead the device 100 has moved that smalldistance to the left. This often happens when the hands holding thedevice 100 shake. Continuing with the above example, in traditionaloperation, the video-stabilization engine 108 attempts to compensate forthis small movement of the device 100 by shifting the image of thesecond frame back to the left. The motion of the device 100 has beencompensated for by moving the image in the succession of captured frames104, and in the resultant video, which comprises the motion-compensatedoutput frames 112, there are, ideally, no lingering effects of theshaking of the device 100.

(As a side note, in order to shift the captured image to compensate forthe slight motion of the personal electronics device 100, the capturedframes 104 are actually somewhat larger than the video eventuallydisplayed. The captured frames 104 include a small extra border ofpixels so that if, for example, the image has shifted to the left andthe leftmost pixels have thus disappeared off the edge of the normalimage, these leftmost pixels are still captured in the left border andcan be used when the image is moved back to the right to compensate forthe motion. This is all well known in the art.)

The motion detector module 110 directly detects physical motion of thepersonal electronics device 100. Various known hardware implementationsare possible here. A three-axis accelerometer can report movement, ascan a gyroscope, or a GPS receiver. In some embodiments, the motiondetector module 110 is based on an image sensor (that is, an imagesensor other than the video camera 102, e.g., a second camera), and itindirectly calculates motion based on perceived motion.

Generally speaking, the personal electronics device 100 includes othercomponents such as a screen for viewing captured images and video, aninterface for receiving user commands and for displaying statusinformation, memory for storing controller software and captured frames,and a communications module for sending and receiving information.Because these components are well known in the art and are not ofparticular concern for the present discussion, they need not bediscussed further.

FIG. 2 shows a first representative method usable by the personalelectronics device 100 of FIG. 1. In step 200, the video-capture engine(or camera) 102 captures and stores a succession of image frames 104.

(Note that for ease of discussion, the representative methods of FIGS.2, 4, and 5 are illustrated and discussed sequentially. In manyembodiments, various steps are performed concurrently, possibly bydifferent processors and controllers working, at least to some extent,independently.)

In step 202, well known techniques of video-stabilization analysis areapplied to the succession of captured frames 104. In one embodiment,successive frames are compared pixel-by-pixel by the video-stabilizationengine 108. If the analysis decides that the differences between framesare best explained, at least in part, by motion of the personalelectronics device 100, then one or more stabilization-compensationvalues are calculated. In some well known embodiments of videostabilization, a calculated stabilization-compensation value minimizes asum of “pixel-error” differences if it were applied to the second of twosuccessive captured frames 104. For side-to-side motion, for example, astabilization-compensation value could be a vector whose scalar value isthe number of pixels that the detected motion represents and thereforethe number of pixels to shift the second image to match (at leastapproximately) the first image to compensate for motion of the device100. In a general case, the analysis can detect motion in all threespatial dimensions as well as rotations. (Note that in step 202, thestabilization-compensation values are calculated but are not applied tothe captured frames 104.)

In many embodiments, the motion detector 110 constantly produces motiondata that represent the detected motions of the personal electronicsdevice 100. In other embodiments, motion data are only produced ifmotion is detected (i.e., no updated motion data mean no detectedmotion).

In any case, the current motion data from step 204 are analyzed in step206. Specifically, the currently detected motion data are comparedagainst a noise threshold value. Most motion detectors 110 known in theart produce some statistically random motion data even when the personalelectronics device 100 is completely at rest. Laboratory tests can berun for a particular motion detector 110 by locking it in place and thenreading its motion-data output. Viewing the output generally clearlyreveals the noise threshold of the motion detector 110. (Of course,statistical techniques more sophisticated than eyeballing a graph can beused to characterize the noise threshold.)

FIG. 3 a shows the results of an actual test of a motion detector 110that was held completely static. The graph of FIG. 3 a shows the motiondata (in three dimensions) produced in this static test case. The graphshows very frequent motion events, all of small amplitude. This ischaracteristic of detector noise. For this particular motion detector110, a review of the graph reveals a noise threshold of plus or minus0.08. (The actual value of the noise threshold for a particular motiondetector 110 is of less importance than the presence of a well definedthreshold. Of course, lower threshold values are generally better, butachieving a very low threshold may involve production costs that faroutweigh any added benefits.)

If the motion data produced in step 204 are below the experimentallyderived noise threshold of the motion detector 110, then the personalelectronics device 100 is most likely at rest. This is the case even ifthe video-stabilization engine 108 produces a non-zerostabilization-compensation value in step 202. That non-zero value ismost likely to be simply incorrect. Therefore, the method of FIG. 2concludes in step 208 by only applying the stabilization-compensationvalue of step 202 to the captured frames 104 if the motion data producedin step 204 are above the noise threshold of the motion detector 110.Otherwise, the device 100 is static, and the stabilization-compensationvalue is simply ignored.

(The question can be asked: Why would a properly functioningvideo-stabilization engine 108 produce a non-zerostabilization-compensation value when the personal electronics device100 is completely static? The answer is based on the realization thatthe video-stabilization engine 108 does not measure motion directly butonly infers motion by comparing successive captured frames 104. Thatanalysis is “fooled” when, to give just one example, more than half ofthe captured image is actually moving smoothly and steadily across theframe (e.g., when a large object near the front of the scene is movingfrom right to left). In this case, most embodiments of thevideo-stabilization engine 108 believe that it is the device 100 that ismoving rather than the majority of image being captured. Thus, itproduces a non-zero stabilization compensation value when it should, inan ideal world, produce a zero value. If that non-zero value wereapplied to the captured frames 104 to compensate for the non-existentmotion of the static device 100 (that is, if the method of FIG. 2 werenot applied), then the resulting output video 112 would include annoying“motion artifacts.” The method of FIG. 2 prevents these artifacts.)

Note: Performing step 202 consumes power and resources within thepersonal electronics device 100 even when the stabilization-compensationvalues are eventually discarded in step 208 because the device 100 isstatic. Thus, in some embodiments, step 202 is only performed if theanalysis of step 206 reveals that the device 100 may actually be moving.In other embodiments, step 202 is always performed, but the results ofthat step are only used when they are appropriate.

Compare FIG. 3 a with FIG. 3 b. (The bold horizontal bars at about plusand minus 0.1 of FIG. 3 b correspond to the noise threshold measured inFIG. 3 a.) The motion data of FIG. 3 b were collected while the personalelectronics device 100 was held in one hand. FIG. 3 b shows veryfrequent motion events, many of which have amplitudes that far exceedthe noise threshold. In this case, video stabilization would be veryuseful, so step 208 of FIG. 2 allows the stabilization-compensationvalues to be applied to the captured frames 104 to create the outputframes 112. (Techniques for applying stabilization-compensation valuesto video are well known in the art.)

The data for FIG. 3 c were taken when the personal electronics device100 was held in two hands. Comparing the amplitudes and the frequency ofthe motion events in this figure with those of FIG. 3 b, it is clearthat a two-hand grip makes for a much more stable device 100. However,the motion amplitudes in FIG. 3 c do still sometimes exceed the noisethreshold, though usually by less than twice the threshold value. Astrict application of the method of FIG. 2 to this situation would allowthe stabilization-compensation values to be applied to the capturedframes 104. However, this is a borderline case, and more data would beuseful to make the best decision about using video stabilization. FIG.4, discussed below, presents one way to do this.

FIG. 4 shows a second representative method usable by the personalelectronics device 100 of FIG. 1. The method of FIG. 4 begins in afashion similar to the method of FIG. 2. Image frames 104 are capturedby the camera 102 (step 400). From the captured frames 104, the videostabilization engine 108 calculates one or morestabilization-compensation values (step 402). Meanwhile, the motiondetector 110 produces motion data (step 404).

In step 406, the method of FIG. 4 diverges from that of FIG. 2. Here,the motion data produced by the motion detector 110 are compared againstthe stabilization-compensation values calculated by the videostabilization engine 108. In high-level terms, this means that twoindependently derived values for the current motion of the personalelectronics device 100 are compared. If the motion detector 110 agrees(within a motion-difference threshold) with the video-stabilizationengine 108 as to how, and how much, the device 100 is currently moving,then their agreed upon value is most likely correct. (Note that, asmentioned above, the motion detector 110 may be implemented as a secondvideo camera, such as a front-facing camera on a laptop computer. Inthis case, step 406 compares the stabilization-compensation values fromthe two cameras. If the value agree within the motion-differencethreshold, then their agreed motion value is most likely to be correct.)In this case, step 408 applies the stabilization-compensation value tothe captured frames 104 to create the output frames 112. If the twomotion values disagree, then the video-stabilization engine 108 isprobably being fooled, so its stabilization-compensation values cannotbe relied upon and are discarded.

Laboratory tests of the personal electronics device 100 can be used toset the value of the motion-difference threshold. In some cases, thenoise threshold derived for the motion detector 110 when the device 100is held static (see FIG. 3 a and the accompanying discussion) may beused for the motion-difference threshold. It is also conceivable thatlaboratory testing can find, for some hardware embodiments, an optimalmotion-difference threshold that is different from the noise threshold.Different hardware embodiments can lead to different results.

Finally, FIG. 5 shows a third representative method usable by thepersonal electronics device 100 of FIG. 1. This method combines aspectsof the methods of FIGS. 2 and 4. As in those methods, image frames 104are captured by the camera 102 (step 500). From the captured frames 104,the video stabilization engine 108 calculates one or morestabilization-compensation values (step 502). Meanwhile, the motiondetector 110 produces motion data (step 504). As in step 406 of FIG. 4,in step 506 the motion data produced by the motion detector 110 arecompared against the stabilization-compensation values calculated by thevideo stabilization engine 108.

Step 508 applies both the test of step 208 of FIG. 2 and the test ofstep 408 of FIG. 4. Thus, only if the personal electronics device 100 isnot static (that is, if the motion data produced in step 504 are abovethe noise threshold of the motion detector 110), and if the motiondetector 110 agrees (within the motion-difference threshold) with thevideo-stabilization engine 108 as to how, and how much, the device 100is currently moving, then the stabilization-compensation values areapplied to the captured frames 104 to create the output frames 112.Otherwise, the video-stabilization engine 108 is probably being fooled,so its stabilization-compensation values cannot be relied upon and arediscarded.

In view of the many possible embodiments to which the principles of thepresent discussion may be applied, it should be recognized that theembodiments described herein with respect to the drawing figures aremeant to be illustrative only and should not be taken as limiting thescope of the claims. Therefore, the techniques as described hereincontemplate all such embodiments as may come within the scope of thefollowing claims and equivalents thereof.

We claim:
 1. On a personal electronics device comprising a camera, adigital video stabilizer, and a motion detector, a method comprising:producing, by the camera, a series of images; calculating, by thedigital video stabilizer, a stabilization-compensation value associatedwith the series of images; producing, by the motion detector, motiondata; analyzing the motion data and the stabilization-compensationvalue; and only if the analyzing reveals that motion detected by themotion detector differs from motion calculated by the digital videostabilizer by less than a motion-difference threshold, then applying thestabilization-compensation value to the series of images.
 2. The methodof claim 1 wherein the personal electronics device is selected from thegroup consisting of: a camera, a mobile telephone, a personal digitalassistant, and a tablet computer.
 3. The method of claim 1 wherein themotion detector comprises an element selected from the group consistingof: a gyroscope, an accelerometer, a GPS receiver, and an image sensor.4. The method of claim 1 wherein calculating astabilization-compensation value comprises comparing images in theseries of images and minimizing a sum of pixel-error differences betweenthe compared images.
 5. The method of claim 1 wherein thestabilization-compensation value is selected from the group consistingof: a linear offset and a rotation.
 6. The method of claim 1 whereincalculating the motion-difference threshold comprises measuring motiondata produced by the motion detector while the personal electronicsdevice is held static.
 7. The method of claim 1 wherein applying thestabilization-compensation value comprises modifying an image in theseries of images, the modifying based, at least in part, on thestabilization-compensation value, and sending the modified image to avideo encoder.
 8. A personal electronics device comprising: a cameraconfigured for producing a series of images; a digital video stabilizerconfigured for calculating a stabilization-compensation value associatedwith the series of images; a motion detector configured for producingmotion data; and a processor operatively connected to the camera, to thedigital video stabilizer, and to the motion detector, the processorconfigured for: analyzing the motion data and thestabilization-compensation value; and only if the analyzing reveals thatmotion detected by the motion detector differs from motion calculated bythe digital video stabilizer by less than a motion-difference threshold,then applying the stabilization-compensation value to the series ofimages.
 9. The personal electronics device of claim 8 wherein thepersonal electronics device is selected from the group consisting of: acamera, a mobile telephone, a personal digital assistant, and a tabletcomputer.
 10. On a personal electronics device comprising a camera, adigital video stabilizer, and a motion detector, a method comprising:producing, by the camera, a series of images; calculating, by thedigital video stabilizer, a stabilization-compensation value associatedwith the series of images; producing, by the motion detector, motiondata; analyzing the motion data and the stabilization-compensationvalue; and only if the analyzing reveals that motion detected by themotion detector is above a noise threshold and if the analyzing revealsthat motion detected by the motion detector differs from motioncalculated by the digital video stabilizer by less than amotion-difference threshold, then applying thestabilization-compensation value to the series of images.
 11. The methodof claim 10 wherein the personal electronics device is selected from thegroup consisting of: a camera, a mobile telephone, a personal digitalassistant, and a tablet computer.
 12. The method of claim 10 wherein themotion detector comprises an element selected from the group consistingof: a gyroscope, an accelerometer, a GPS receiver, and an image sensor.13. The method of claim 10 wherein calculating astabilization-compensation value comprises comparing images in theseries of images and minimizing a sum of pixel-error differences betweenthe compared images.
 14. The method of claim 10 wherein thestabilization-compensation value is selected from the group consistingof: a linear offset and a rotation.
 15. The method of claim 10 whereincalculating the noise and motion-difference thresholds comprisesmeasuring motion data produced by the motion detector while the personalelectronics device is held static.
 16. The method of claim 10 whereinapplying the stabilization-compensation value comprises modifying animage in the series of images, the modifying based, at least in part, onthe stabilization-compensation value, and sending the modified image toa video encoder.
 17. A personal electronics device comprising: a cameraconfigured for producing a series of images; a digital video stabilizerconfigured for calculating a stabilization-compensation value associatedwith the series of images; a motion detector configured for producingmotion data; and a processor operatively connected to the camera, to thedigital video stabilizer, and to the motion detector, the processorconfigured for: analyzing the motion data and thestabilization-compensation value; and only if the analyzing reveals thatmotion detected by the motion detector is above a noise threshold and ifthe analyzing reveals that motion detected by the motion detectordiffers from motion calculated by the digital video stabilizer by lessthan a motion-difference threshold, then applying thestabilization-compensation value to the series of images.
 18. Thepersonal electronics device of claim 17 wherein the personal electronicsdevice is selected from the group consisting of: a camera, a mobiletelephone, a personal digital assistant, and a tablet computer.